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Abstract 

The present study aimed to test the hypothesis that a smiling expression on the face of a talking 
pedagogical agent could positively affect a learner’s emotions, motivation, and learning outcomes in a 
virtual learning environment. Contrary to the hypothesis, results from Experiment 1 demonstrated that 
the pedagogical agent’s smile induced negative emotional and motivational responses in learners. 
Experiment 2 showed that the social meaning of a pedagogical agent’s smile might be perceived by 
learners as polite or fake. In addition, qualitative data provided insights into factors that may cause 
negative perceptions of a pedagogical agent’s smile, which in turn lead to negative affective (emotional 
and motivational) states in learners. Theoretical and design implications for pedagogical agents in 
virtual learning environment are discussed in the concluding section of the paper. 
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Introduction 

One of the major drawbacks of e-learning relates to the lack of social and personal presence in this 
environment (Searls, 2012). This issue has prompted the implementation of artificial intelligent 
characters known as “pedagogical agents” in virtual learning systems (Kramer & Bente, 2010; 
Veletsianos & Russell, 2014). Pedagogical agents are visual characters that reside within the digital 
space of virtual learning systems. They are viewed as intrinsically social interfaces, which allows for the 
inclusion of socio-emotive values in the learning process (Guo & Goh, 2015; Kim & Baylor, 2015). 
Embedded within virtual learning environments, pedagogical agents take instructional roles such as 
tutor, learning companion and coach in e-learning environments (Kim & Baylor, 2006; Kim & Baylor, 
2015). 

Proponents of pedagogical agent have claimed that the visual embodiment of an agent can create a 
“persona effect” in virtual educational systems. The persona effect refers to a phenomenon in which the 
mere presence of an agent leads to an enhanced learning experience in a virtual learning environment 
(see Lester et al., 1997; Moreno, Mayer, Spires, & Lester, 2001). Moreover, it was also argued that 
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pedagogical agents can help induce a sense of social cues in a multimedia message that can prime the 
social conversation schema in learners (Mayer, Sobko, & Mautone, 2003). 

Against this background, advocates of pedagogical agents have posited that if visual interfaces of agents 
are integrated into learning systems, learners will experience higher motivation to learn (Schroeder, 
Adesope, & Gilbert, 2013). In a similar vein, an agent may function as a “social model” that can influence 
a learner’s motivation and attitude regarding a subject domain (Plant, Baylor, Doerr, & Rosenberg- 
Kima, 2009; Rosenberg-Kima, Baylor, Plant, & Doerr, 2008; Yilmaz & Kihq-Cakmak, 2012). Previous 
studies have shown that pedagogical agents that were young, female and “cool” were instrumental in 
positively changing young women’s attitudes and self-efficacy in engineering-related domains (Plant et 
al., 2009; Rosenberg-Kima et al., 2008). 

In addition, the inclusion of a pedagogical agent may allow relevant information in virtual environment 
to be pointed out effectively during the learning process. For instance, prior studies have demonstrated 
that participants learned more in a system with a pedagogical agent than in a system without an agent 
(Atkinson, 2002; Yung & Paas, 2015). Atkinson (2002) suggested that the agent enhanced learning by 
providing gestures that helped learners to pay attention to relevant information. With the presence of 
an agent, learners’ cognitive resources were effectively utilized, as learners did not need to dedicate their 
mental efforts to searching the virtual learning space for relevant visual information (Yung & Paas, 2015, 
Atkinson, 2002). 

Pedagogical agents can also function as “vessels” that contain distributed information (Veletsianos & 
Russell, 2014). In other words, pedagogical agents can scaffold learners by asking questions, providing 
hints, or offering alternative perspectives during learning. Holmes (2007) demonstrated that learners 
who interacted with pedagogical agents produced deeper explanations in post-tests than learners who 
did not interact with an agent. In his paper, Holmes pointed out that the agents were designed to 
stimulate deeper understanding in novice learners by listening and providing feedback regarding the 
subject domain. Hence, learners who “conversed” with agents had the advantage of exercising good 
explanatory skills and, hence, outperformed learners who did not interact with the agents. 

In virtual learning systems, can a learner’s emotional and motivational response be affected by the facial 
expression of a virtual agent? If so, what kind of emotional response is induced by what type of agent’s 
expression? These are interesting questions, as recent advances in multimedia technology allow the 
development of realistic virtual learning agents with emotional states indicated through speech 
(synthetic and/or recorded), body language, and facial expressions (Beale & Creed, 2009). As more and 
more embodied agents are being implemented in educational virtual domains (i.e., virtual learning, e- 
training, serious gaming, edutainment and blended learning), it is hypothesized that the design of an 
agent’s facial expression has a strong impact on learner-virtual agent interaction. 

Nevertheless, it is still unclear how people perceive and respond to synthetic displays of emotions by 
virtual agents (Beale & Creed, 2009). Against this background, the present study aims to explore the 
effects on emotional, motivational and learning outcomes in learners as a result of perceiving facial 
expressions of virtual learning agents. More specifically, we focus our investigation on how a 
pedagogical agent’s smiling expression affects the learner’s emotions, motivation and learning 
outcomes. 

The Effects of an Agent's Smile in Virtual Applications 
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The “media equation” makes a compelling case that human-to-human rules of social psychology are 
also relevant to human-virtual agent interaction (Reeves & Nass, 1996). Studies have shown that the 
social rules that are established in human-to-human communications are also prevalent in human-to- 
computer settings. Some of the social interplays that have been observed empirically in human- 
computer experiments are the stereotype effect (Liew, Tan, & Jayothisa, 2013), inhibition/facilitation 
effect (Hayes, Ulinski, & Hodges, 2010; Zanbaka, Ulinski, Goolkasian, & Hodges, 2007), politeness 
effect (Wang et al., 2008), similarity attraction (Nass & Lee, 2001), empathy (Guadagno, Swinth, & 
Blascovich, 2011) and the venting effect (Klein, Moon, & Picard, 2002). 

Emotional contagion refers to the automatic tendency for a person’s emotional state to be affected by 
another person’s emotional expression (Hatfield, Cacioppo, & Rapson, 1994). In social interaction, 
people unconsciously send cues about their emotional state through non-verbal signals such as facial 
expressions, vocal inflections, body gestures, and posture, and perceive such cues sent by others. The 
inducement of emotions between people is not caused by an intended action of a person, but only by 
the perception of the emotional expression of that person (McIntosh, Druckman, & Zajonc, 1994). 
Literature has shown that emotional contagion can also occur in human-computer interaction (de Melo, 
Carnevale, & Gratch, 2012; Kramer, Kopp, Becker-Asano, & Sommer, 2013; Neumann & Strack, 2000; 
Qu, Brinkman, Ling, Wiggers, & Heynderickx, 2014; Wong & McGee, 2012). As discussed below, smiling 
expressions of agents have been shown to affect users’ emotional and motivational experiences in virtual 
systems. 

In one study, Tsai, Bowring, Marsella, Wood, and Tambe (2012) showed that participants experienced 
increased happiness simply by viewing static images of smiling virtual agents. Tsai and his team 
conducted another experiment to ascertain if the effect of emotional contagion would positively impact 
participants’ trust and likelihood of cooperation in a virtual simulation of strategic decision making. 
However, it was found that the effect of emotional contagion disappeared in the virtual scenario. In a 
subsequent experiment, the emotional contagion effect reappeared (albeit not reaching statistical 
significance) when the need for decision making was removed from the virtual scenario. Taken together, 
Tsai and his team proposed that participants’ act of deliberate thinking (as required to make a strategic 
decision) diminished the effect of emotional contagion. 

Related research by Ku et al. (2005) revealed that participants felt congruent emotions while observing 
virtual agents with different facial expressions agents (i.e., neutral, happy, and angiy). Interestingly, 
they demonstrated that the intensity of congruent emotions felt by viewers increased according to the 
intensity of the facial expressions by virtual avatars. This observation is in line with the mechanism of 
mimicry, in which greater emotional display by the agents results in higher levels of emotional 
contagion in people. Though the study did not extend its findings to virtual applications and scenarios, 
it asserted that people’s emotional responses are spontaneously affected by facial expressions of virtual 
agents. 

In an experiment by Kramer et al. (2013), participants were paired with either a non-smiling agent, an 
infrequent-smiles agent, or a frequent-smiles agent. Though the frequency of agent smiles did not 
differently affect participants’ perceptions of the agent, it was shown that participants tended to mimic 
the smile of the virtual agent. Furthermore, it was observed that these mimicry responses were elicited 
even when participants did not consciously notice the facial expression of the virtual agent. Though 
participants’ emotions were not measured in the study, the study suggested the mechanism of facial 
mimicry in human-agent interaction. 
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An experiment by Pertaub, Slater, and Barker (2002) showed that speakers in virtual environment were 
affected emotionally by a virtual avatar’s non-verbal characteristics such as facial expressions, body 
postures, and gestures. As predicted, speakers felt less anxiety when paired with virtual audiences that 
displayed positive expressions such as encouraging nods, frequent smiles, and appropriate body 
orientation. In contrast, virtual audiences that showed negative characteristics such as yawning, 
slouching, and avoidance of eye contact induced a higher level of anxiety response from the human 
speaker. 

De Melo et al. (2012) demonstrated that the virtual agents’ static facial displays of joy, sadness, anger 
and guilt influenced participants’ decision to counter-offer, accept or drop out from virtual negotiations. 
Furthermore, people’s expectations about the agents’ decisions were impacted by these facial 
expressions. Though there were no measures of emotion taken from the participants, the study affirmed 
that the facial expressions of virtual agents induce subconscious behavioral and cognitive responses 
from people. 

Research Motivation 

Based on these related works, there is empirical evidence regarding the effects of smiling expressions 
by agents in human-agent interaction. More specifically, there is a tendency for people to automatically 
“catch” emotions from smiling expressions of virtual agents (Kramer et al., 2013; Ku et al., 2005; Tsai 
et al., 2010). These findings seem to support the emotional transference mechanism, in which people 
are more likely to experience emotional convergence (rather than divergence) by simply observing the 
facial expressions of virtual agents. Studies regarding agent’s smile have been conducted in virtual 
scenarios such as decision-making (Tsai et al., 2012), negotiation (de Melo et al., 2012) and games 
(Kramer et al., 2013). 

However, research related to the effects of an agent’s smile in a virtual learning environment is still 
scant. This research gap needs to be filled, as there is a potential for virtual agents to express motivation 
and excitement in a way that is infectious to human learners (Beale & Creed, 2009; Burleson, Picard, 
Perlin, & Lippincott, 2004). In other words, it might be interesting to determine if an agent’s smile can 
automatically induce positive emotions in learners. It is further hoped that the induced positive emotion 
can have a facilitating effect on learners’ motivation and perception of learning environments (Liew & 
Tan, 2016; Isen & Reeve, 2005; Um, Plass, Hayward, & Homer, 2012; Yusof & Zin, 2013). Hence, we 
hypothesized that an agent’s smiling expression will positively enhance a learner’s motivation, 
perceptions and learning outcomes in a virtual learning environment. 

In addition, prior studies tested the effects of an agent’s smile with static images of virtual agents (see 
de Melo et al., 2012; Ku et al., 2005; Tsai et al., 2012). As there is an increasing trend and expectation 
for believable agents to dynamically express emotions in virtual applications, it is, therefore, imperative 
to extend this line of investigation with animated virtual agents. Previous work has only explored the 
effects of an agent’s smile with the agent’s facial expression as the sole modality and source of emotional 
cues (see de Melo et al., 2012; Ku et al., 2005; Tsai et al., 2012). The effect of a smiling expression has 
not yet been tested for scenarios in which facial expressions and vocal expressions are presented by 
talking virtual agents. 

Against this background, this paper aims to extend the field by addressing two key points. Firstly, we 
aim to investigate the effects of smiling expression with talking pedagogical agents rather than static 
images of pedagogical agents. That is, we aim to use agents that display dynamic facial expressions (i.e., 
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eye, lips, head, and facial movement), which are synchronized with computer-generated speech. Similar 
to previous studies (Ku et al., 2005; Tsai et al., 2012), we focus on smiling expressions of talking virtual 
agents as the potential cause for emotional contagion in learner-agent interaction in virtual learning 
systems. Secondly, we aim to explore the effects of an agent’s smiling expression in the context of a 
virtual learning environment. 

Pursuant to this goal, two experiments were conducted in which participants were either exposed to a 
neutral or a frequent smiling expression of a virtual agent. Experiment 1 explored the effects of an 
agent’s smiling expression on the learner’s emotions, motivation and learning outcomes. Experiment 2 
served as a post-hoc study that utilized quantitative and qualitative data to assess how learners might 
perceive the meaning of smiles by the virtual agent in the virtual learning environment. 

Experiment 1 

In this experiment, we aimed to determine the effects of an agent’s smiling expression on a learner’s 
emotions, motivation, and learning outcomes in virtual learning environments. This experiment also 
served to test the hypothesis that a smiling expression of a virtual agent will positively affect a learner’s 
emotions, motivation and learning outcomes. 

Participants 

One hundred and seven freshmen (40 males, 67 females, mean age = 20) at a university were recruited 
to participate in exchange for extra course credits. All participants were business majors who had no 
experience in programming-related subjects (including virtual learning) and did not have any 
experience interacting with virtual agents. 

Virtual Agents in Virtual Learning Environments 

Two 3D male virtual agents were designed to convey either a neutral facial expression or a constant 
smiling expression (see Figure 1). The audio speech of both agents was generated using the IVONAText- 
To-Speech engine (American English-Joey, default speed). The virtual learning environment served as 
an interactive platform for the virtual agent to deliver lessons on basic computer programming (see 
Figure 2). 



Figure 1. Neutral-face agent and smiling-face agent. 


252 



The Effects of a Pedagogical Agent’s Smiling Expression on the Learner’s Emotions and Motivation in a Virtual Learning Environment 

Liew, Zin, Sahari, and Tan 



Figure 2. Virtual learning environment. 

Dependent Measures 

To measure learners’ emotions, the positive affect scale from PANAS-X was utilized (Watson & Clark, 
1999). The survey asked respondents to indicate the degree to which they experienced 10 types of 
positive feelings using a 5-point Likert-type scale ranging from 1 (very slightly or not at all) to 5 (very 
much) (coefficient alpha = .84). The total score for General Positive Affect was obtained by adding the 
10 responses to the items. Besides the positive affect, the different subscales obtained from the survey 
were joviality, self-assurance, and attentiveness (see Watson & Clark, 1999). 

A survey consisting of three items, each scaled from 1 (strongly disagree) to 7 (strongly agree) measured 
learners’ perceptions of the virtual lessons: 1) the lesson is fun; 2) the lesson makes me curious to know 
more about the subject; and 3) I am willing to come back to this lesson in the future. Learners’ 
perceptions of virtual agents were measured by 5 items, each scaled from 1 (strongly disagree) to 7 
(strongly agree): (1) I like Michael (the virtual avatar), (2) Michael is knowledgeable, (3) Michael is 
friendly, (4) I can trust Michael, and (5) I am willing to learn with Michael in the future. 

Learning achievements of participants were measured as total scores on comprehension post-tests. The 
comprehension test asked learners to predict the output of twelve short program statements. The 
transfer tests required learners to apply algorithmic rules as learned in the virtual lessons to the new 
situation, in order to achieve the correct output. Each correct output was awarded 1 point; hence, the 
maximum score obtainable was 12. These measures were consistent with the cognitive skills of 
knowledge retention and comprehension in computer science. 

Procedure 

Participants (n = 107) were randomly paired with the neutral virtual agent (control group, n = 53) or 
smiling virtual agent (experimental group, n = 54). All participants were seated in front of individual 
computers with earphones. To deter participants from suspecting the intention of the experiment and 
thus paying attention to the facial expression of the virtual agent and to their own emotions, two 
important measures were taken: (1) participants in the control and experimental groups were separated 
accordingly into two different laboratories so that they would be unaware of the different conditions, 
and (2) participants were told that the study was for the purpose of measuring computer aptitudes. After 
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the 30-minute interactive lesson with the virtual agent, participants were instructed to fill up the 
surveys (in order) for measuring emotions, perceptions on the virtual learning, and perceptions on the 
virtual agent. Finally, participants were given 30 minutes to complete the post-test. 

Results 

An independent sample t-test was performed to compare scores for emotion, perception of virtual 
learning, perception of the virtual agent, and the post-test between participants in smiling- and neutral- 
agent conditions. Table 1 shows the means, standard deviations and p-values of the t-test analysis. 

Table 1 

Means, Standard Deviations and p-values of t-test Analysis 


Control (Neutral) Experimental p-value (2- 

[Mean, SD] (Smiling) tailed) 

[Mean, SD] 


Emotion 


General Positive Affect* 

28.36 (7.77) 

25-67 (7-53) 

< 0.1 

Joviality* 

8.42 (2.58) 

7-43 (2.79) 

< 0.1 

Self-Assurance** 

5-49 (1-89) 

4.70 (1.84) 

< 0.05 

Attentiveness 

8.49 (2.58) 

8.14 (2.81) 

> 0.1 


Perception of learning 
environment 


Enjoyment 

3-92 

(1-63) 

3-79 (i-34) 

> 0.1 

Curiosity** 

4-43 

(1.51) 

3.88 (1.60) 

< 0.05 

Willingness to return* 

3-84 

(i-7i) 

3-33 (i-53) 

= 0.05 
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Perception of virtual agent 




Likability 

4-15 (i-43) 

4.16 (1.60) 

> 0.1 

Knowledgeable 

4-77 (1-50) 

4.71 (i-66) 

> 0.1 

Credibility 

4-77 (i-37) 

4.70 (1.65) 

> 0.1 

Friendliness 

4-49 (i-59) 

4-73 (1.61) 

> 0.1 

Willingness to return 

3.81 (1.67) 

4.21 (1.43) 

> 0.1 

Post-test 

7-13 (3-05) 

7.71 (3-17) 

> 0.1 


* significance at p < o.i 
** significance at p < 0.05 

As shown in the table, participants experienced marginally less positive emotion when paired with the 
smiling virtual agent, 1(105) = 1-755, P = 0.072. Based on the positive affect sub-scales, it was shown 
that the smiling virtual agent had a negative impact on participants’ feelings of joviality, 1(105) = 1.820, 
p = 0.082 and self-assurance, 1(105) = 2.184, P = 0.031. Finally, the smiling virtual agent had a negative 
impact on learners’ perceptions of virtual learning. It was shown that the participants experienced lower 
curiosity to explore the virtual lesson more, 1(105) = 225,41 = 0.028 and less willingness to return to the 
virtual lesson, 1(105) = 1-974, P = 0.05 than did participants engaging the neutral agent condition. No 
significant effects were found for participants’ perceptions of the virtual agents and post-test scores. 
Finally, results from 2-way ANOVA determined that these findings were not moderated by participant’s 
gender; for all significances, p > 0.05. 


Discussion 

The purpose of this study was to determine the effects of agent’s smiling expression on learner’s 
emotion, motivation and learning outcome in virtual learning systems. To achieve this, participants 
were either paired with the neutral-face agent or smiling-face agent in a virtual learning environment. 
Based on the data, there were no significant differences of post-test scores between learners in smiling 
and neutral agent conditions. Hence, learning outcome was not affected by the presence of a smiling 
agent. 

However, it was shown that participants who interacted with the smiling-face virtual agent experienced 
lower positive affect, joviality and self-assurance than did participants who interacted with the neutral- 
face virtual agent. Furthermore, the smiling expression of the virtual agent eroded participants’ 
perceptions of virtual learning environments. This discovery is at odds with earlier findings (Ku et al., 
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2005; Tsai et al., 2012), which highlighted that the smiling expression of virtual agents causes congruent 
(i.e., joyful) emotions in participants. 

Why did the smiling expression of the agent negatively impact learners’ emotions and motivation in 
virtual learning systems? It is plausible that participants’ emotional responses to a smiling expression 
of the virtual agents are different, based on participants’ diverse appraisals of the perceived meaning of 
smiles. Indeed, recent research posits that people distinguish between genuine and fake smiles given by 
virtual agents (Ochs, Niewiadomski, & Pelachaud, 2012; Ochs & Pelachaud, 2013). In particular, 
genuine smiles (also called felt or Duchenne smiles) express an agent’s emotional state of happiness, 
while fake smiles communicate politeness, embarrassment, or hiding true feelings (Ochs & Pelachaud, 
2013; Rliem & Andre, 2005). In fact, an investigation of human-virtual agent interaction showed that 
people’s feelings of enjoyment are affected by the perceived meaning of virtual agents’ smiles (Ochs & 
Pelachaud, 2013). 

To distinguish the dissimilar meanings of smiles, we follow an emergent theory proposed by Niedenthal, 
Mermillod, Maringer, and Hess (2010) called the Model of Simulation of Smiles (SIMS). In this model, 
three types of smiles are proposed: ones to indicate felt positive emotion (enjoyment smiles); ones to 
express social motives, such as politeness and greetings (affiliative smiles); and smiles to communicate 
dominance, pride, and masking of negative intentions (dominant smiles). Placing the SIMS in human- 
virtual agent interaction, it seems likely that the emotion-congruence effect (i.e., positive emotion) 
occurs only if virtual agents are perceived to be expressing enjoyment smiles. Conversely, emotion- 
divergence (i.e., negative emotion) is induced when smiles of virtual agents are perceived as fake, such 
as a dominant smile that masks negative emotions. 

Following this view, we hypothesize that the negative emotion and reduced motivation occurred 
because participants might have perceived the virtual agent’s smile as fake (i.e., dominant) rather than 
genuine smile (i.e., affiliative or enjoyment). To seek a better understanding regarding this issue, a 
follow-up study (Experiment 2) was conducted. Experiment 2 served to determine how the smiling 
expression of the virtual agent might be perceived by learners. Based on the SIMS, the prototypical 
smiles under investigation are enjoyment, affiliative, and dominant. A qualitative survey was also 
utilized to shed light on learners’ perceptions of a virtual agent’s smile. 

Experiment 2 

The outcomes of Experiment 1 showed that the smiling expression of the virtual agent had negative 
impacts on participants’ emotions and motivations. Based on the factors discussed above, we 
hypothesized that the emotion-divergence occurred because participants were more likely to perceive 
the agent’s smile as dominant rather than enjoyment/affiliative. Hence, this study examined how the 
smiling expressions of pedagogical agents are interpreted by learners in a virtual learning system. 

Participants 

From the same university as in Experiment 1, 56 freshmen (23 males, 33 females, mean age = 19) were 
recruited to participate in exchange for partial course credits. No participant had had any previous 
experience with virtual agents. 

Materials 
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For the purpose of this study, we used the virtual learning environment that is paired only with the 
smiling-face agent (see Figures 1 and 2). 

Dependent Measures 

Based on the SIMS, a multiple choice survey asked participants to choose one of the following: the 
virtual agent is smiling because he is happy (enjoyment); the virtual agent is smiling because he is being 
polite (affiliative); and the virtual agent is smiling because he is hiding his true emotions (dominant). 

Procedure 

Participants (n = 56) were told that the study required them to evaluate the meaning of the smile 
displayed by the virtual agent in the virtual learning environment. Within a closed classroom, 
participants viewed and listened to the lesson presented by the virtual agent, which was projected on a 
4000mm x 4000mm screen. The agent’s speech was channelled through four mounted PA speakers. 
After observing the virtual agent, the participants were asked to select one answer that described their 
perception of the agent’s smile: enjoyment, affiliative, or dominant. Next, they were instructed to give 
their subjective evaluation of the smile displayed by the virtual agent. 


Results and Discussion 

In our earlier discussion, we noted that the social meaning of a pedagogical agent’s smile might be 
interpreted differently by learners. Arguably, negative emotion and motivation may arise because 
learners perceive the smile of a virtual agent as non-enjoyment (dominant or affiliative) rather than 
enjoyment. To explore these issues, a descriptive analysis was performed to compare the frequencies of 
participants’ perception of the three smiles: enjoyment, affiliative, and dominant. As shown in Figure 
3, the perception of an enjoyment smile is o % percent. This means that none of the participants thought 
that the virtual agent was genuinely expressing a state of happiness. Instead, the virtual agent’s smile 
was perceived to be either affiliative/being polite (60.38 %) or dominant/hiding emotions (39.62 %). 
To explore whether gender moderated the perceptions of the agent’s smile (i.e., dominant versus 
affiliative), a Chi-square analysis was conducted. Results indicated that male and females did not 
significantly differ in their perceptions of the agent’s smile, x(t) = 0.04, p > 0.05. 



Figure 3. Perception of agent’s smiles. 

We conducted an exploratory analysis to tease apart the remarks made by participants on the agent’s 
smile (Table 2). Interestingly, there were consistent and recurring themes in the remarks, based on how 
smiles were perceived (affiliative or dominant). Specifically, participants who perceived the smile as 
affiliative commented, “...somehow his smile makes me less stressed,” “less scared because of his smile,” 
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“he is teaching patiently and politely,” “not scared because...smile is so friendly and has true feelings,” 
and “audience will feel comfortable when seeing his smile.” 

Taken together, this implies that an affiliative smile by virtual agents seems to induce positive feeling 
of social attachment and intimacy in participants (as suggested in the SIMS). As for when the agent’s 
smile was perceived as dominant, participants remarked, “creepy and fake,” “forced smile,” “he tries to 
be polite, feel scared because he has no feeling,” “scared...unnatural smile,” “...is hiding his true 
feelings,” “feels like he is going to kill me, his smile is tricky,” “looks happy but fake, feeling scared as 
he is hiding something from audience,” and “he is showing off.” Overall, these comments reflect a sense 
of distrust, withdrawal, vigilance, and general negative emotion in participants who perceived the 
dominant smile. 

Table 2 

Learners’ Comments Regarding Pedagogical Agent’s Smile 


Interpretation of agent’s smiling 

expression 

Comments regarding agent’s smile 

Affiliative (polite) 

• 

somehow his smile makes me less 

stressed 


• 

less scared because of his smile 


• 

he is teaching patiently and politely 


• 

not scared because...smile is so friendly 

and has true feelings 


• 

audience will feel comfortable when 

seeing his smile 


• 

smiling and polite 


• 

he makes me want to listen to him 


• 

he got nice smile and explain about the 

pass system 
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• he makes me more interested on him 
and feel like he acts like open mind, 
optimistic person 

• I feel Michael move timely, polite and 
happy with him studying together 

• friendly seems like kind person 

• Very happy when looking at Michael’s 
smile, not scared because he is polite 

• He made me smile and he looks like my 
dad 

• Look very friendly and happy to be 
teaching C programing 

• He let me feel interesting to learn 
something 

• manners and respects, smile feels 
friendly 


Dominant (fake) 


• creepy and fake 

• forced smile 

• he tries to be polite, feel scared because 
he has no feeling 

• scared...unnatural smile 

• is hiding his true feelings 

• feels like he is going to kill me, his smile 
is tricky looks happy but fake, feeling 
scared as he is hiding something from 
audience 

• he is showing off 
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• agent voice is “emotionless”, “boring,” 
and “monotonous”. 

• agent’s face was “creepy” because of the 
“lack of synchronization between mouth 
movement and speech,” 

• he tries to be polite, feel scared because 
he has no feeling, not really like him 
because he talks with himself 

• The smile is not expressing correctly; 
Michael’s voice indicate that he is not 
feeling happy. The expression on face 
have to be more realistic, muscle 
movement on face the lip-sync with 
voice 

• his smile is not natural feel scared when 
looking at him 

• he keeps smile when talking, he need to 
talk normally rather than acting fake 

• Scared of him because he talks very 
seriously and serious tone, while he 
looks kind of happy 

• very fake I feel scared because he is not 
a true person 

• So robotic 


General Discussion and Conclusion 

This study proposes two key insights that contribute to the research in learner-pedagogical agent 
interaction in virtual learning environments. Firstly, this study shows that facial expression in talking 
virtual agents can automatically and subconsciously induce emotional and motivational responses in 
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learners. The emotion contagion effects were observed even when the smiling expression of the virtual 
agent was not consciously attended to (see procedure of Experiment l). Moreover, our study shows that 
facial expressions in talking agents affect how people perceive virtual environments. Plausibly, 
emotional responses induced by facial expressions of agents changed the way participants perceived the 
virtual environment. Notably, exploratory analysis yields a strong correlation (r > 5, p = 0.000) in all 
relationships between emotion scores and perceptions of the learning environment (i.e., curiosity to 
explore the virtual lesson, willingness to come back to the virtual lesson, and enjoyment of the virtual 
lesson). 

Secondly, our study reveals that the virtual agent’s smile induced a divergent (negative) effect on 
participants’ emotional and motivational states. The effect of emotional divergence was initially 
surprising, as one might expect that the agent’s smile would produce congruent (i.e., feeling of 
happiness) emotions in participants (see Ku et al., 2005; Tsai et al., 2012). However, a deeper 
perspective regarding this issue was found in the literature describing the role of social comparison 
mechanisms in divergent emotional contagion (Epstude & Mussweiler, 2009). In the light of social 
comparison, people do not always perceive an agent’s smile as enjoyment, which expresses a genuine 
state of happiness (Ochs & Pelachaud, 2013). Rather, as shown in Experiment 2, the agent’s smile could 
be interpreted as non-enjoyment, such as affiliative (being polite) or dominant (masking negative 
emotions). It can be further argued that the social interpretation of an agent’s smile as “dominant” and 
“fake” produces negative impacts on emotion and motivation in virtual learning systems. 

Our results from Experiment 2 showed that participants’ interpretations of agent smiles were almost 
equally divided between dominant and affiliative. Curiously, none of the participants perceived the 
agent’s smile as enjoyment. Hence, one might ask: Why was the agent’s smile perceived as non¬ 
enjoyment (dominant or affiliative) rather than enjoyment? Insights regarding this question can be 
found in participants’ comments about the virtual agent and its smile. Some of the remarks regarding 
the agent’s smile and speech (from Experiment 2) indicated that the agent’s speech was emotionless 
and monotonous, albeit being clear and articulate. Hence, the lack of emotion in the agent’s voice 
diminished the perceptions of smiling agents as being truly happy. Instead, the incongruent pairing of 
emotionless voice and a smiling facial expression might have induced a disconcerting emotional effect 
in participants, resulting in various interpretations, such as “fake,” “scary,” “hiding its true emotion,” 
“unnatural,” and “tricky” (Experiment 2). 

This observation mirrors earlier findings by Creed and Beale (2008), which indicated that mismatched 
facial and voice expressions elicit negative responses from people. Furthermore, there were a few 
comments describing the lack of synchronization between speech and lip movement, such as “fake” and 
“creepy,” suggesting the uncanny valley effect, which might have contributed to the negative perceptions 
of the agent’s smile (see Tinwell, Grimshaw, Nabi, & Williams, 2011). Taken together, this study suggests 
that the voice expression and lip synchronization influence the perception of the smile of a talking 
animated virtual. It is worth noting that learners who perceived the agent smile as affiliative (being 
polite) mentioned positively that they were more “comfortable” and “less scared and stressed” when 
engaging the virtual agent (Experiment 2). Similarly, findings in a recent study (Guadagno et al., 2011) 
suggest that an agent’s smile positively impacts participants’ social evaluation of the virtual agent (i.e., 
perceived empathy). 

Based on this prior discourse, it is imperative for virtual agent designers to be aware that a mismatch of 
the facial expression and voice expression of a virtual agent may elicit unfavourable emotional response 
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from learners. In the case of this study, the neutral (emotionless) voice of the virtual agent distorted the 
perception of the virtual agent’s smile, which led to negative emotional and motivational responses from 
learners. As voice quality is shown to impact perception, our future work will include an agent that 
displays congruent emotional cues through both facial and voice expression. These future studies are 
motivated by the quest to determine if congruent positive display of an agent’s facial and vocal 
expression (i.e., happiness, enthusiasm and excitement) can positively influence learners’ affective 
factors such as emotion, motivation and learning perception in a virtual learning environment. 

Limitations and Suggestions for Future Research 

This study is not without limitations. In the context of our experiment, the duration of the virtual lesson 
was restricted to 30 minutes. This duration might not represent the length of a typical lesson in a virtual 
learning environment. Hence, the effects of an agent’s smile have yet to be ascertained for a prolonged 
virtual lesson. Future research should use longitudinal research to enhance understanding of how a 
smiling agent affects cognitive and socio-emotive behaviours in respondents. 

One of the reviewers of this paper pointed out that a participant’s demographic (e.g., ethnicity, gender) 
and personality traits might have impacted his or her perception of an agent’s smile. Unfortunately, our 
study design and data did not allow for additional insights into why some learners perceived the agent’s 
smile as affiliative, while others perceived the agent’s smile as fake/dominant. Hence, future research 
should explore how a learner’s demographic, cultural and personality markers may influence his or her 
perception of an agent’s smile. Moreover, as suggested by the same reviewer, the findings of our study 
should be replicated with different agent designs (i.e., gender, role and ethnicity), in order to associate 
the effects of an agent’s smile with agent properties such as gender (Ozogul, Johnson, Atkinson, & 
Reisslein, 2013; Wang & Yeh, 2013), stereotype (Liew, Tan, & Jayothisa, 2013; Veletsianos, 2010) and 
ethnicity (Kim & Wei, 2011; Moreno & Flowerday, 2006). Last but not least, a control group using the 
voice paired with static images of a smiling agent can be included in future experiments, in order to 
reinforce the validity of the current findings of this paper. 
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